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Research Article 

Dental training in evidence-based 
dentistry: A replication study 


Abstract 

Background: In the area of education research, it is well-known that studies of a defined question 
are seldom replicated. Furthermore, e-learning resources with evidence-based content in dentistry have 
received relatively little attention from researchers. 

The Context and Purpose of the Study: The aim of this clinical study was to evaluate how dentistry 
students from two consecutive cohorts in their first clinical semester rate a long-standing evidence- 
based dentistry (EbD) resource in an e-learning environment using a questionnaire of 43 specific items 
on 1) general questions regarding user-friendliness and acceptability, as well as 2) specific questions on 
content and functional range (A), handling and technical aspects (B), and didactics and educational value 
(C) based on a Likert scale from 0 = 'strongly disagree’ to 3 = 'strongly agree’, and how this compares to 
a primary study in which the resource was addressed as a novelty. The data were analyzed statistically 
using a one-way ANOVA followed by a Kruskal-Wallis multiple-comparison Z-test. 

Results: A response rate of 100% was achieved. The majority of the users thought the topic of EbD 
to be important. The e-learning resource was rated with a score of 2.40 ± 0.66 (on a Likert scale from 
1-6 where 1 = "very good" and 6 = "insufficient"). 86.15% of the students stated that they consider the 
resource beneficial for their study in clinical simulation and in patient treatment courses. The results 
averaged for A: 1.92 (±0.57; median: 1.928), B: 1.48 (±0.60), and C: 2.27 (±0.67). The obtained results in the 
replication study showed no statistical significant differences to the primary study. 

Conclusions: The e-learning resource with dentistry vignettes cases and learning components on 
evidence-based principles was consistently rated positively by the students. Owing to their agreement 
with the data of the primary study, the results of the present study point to the remarkable validity of the 
method of evaluation. This should be addressed in future studies with larger cohorts. 


available or that numerous other parameters are intentionally 
or unintentionally altered in the replication. Concluding 
their recent evaluation, Gerhardt-Szep and colleagues came 
to recommend the e-learning environment in an e-learning 
resource on evidence-based dentistry in a cohort of dental 
students [6]. A modified method of evaluation formed the 
basis of this commendation [7,8]. On behalf of the German 
Association of Medical Informatics, Biometry and Epidemiology 
(GMDS), the authors have coordinated the development of 
a catalogue of “Quality Criteria for Electronic Publications 
in Medicine” (QCEPM) [7]. This catalogue is subdivided into 
categories including contents, technical aspects, and didactics. 
According to these items, typical faults and deficiencies of 
medical electronic software and programmes are elucidated 
and possible solutions are given. These criteria are intended 
to support the formative evaluation during the development 
of each kind of electronic publication, including eLearning 
medical programmes, and to provide a basis for their summative 
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Introduction 

In the area of education research, it is well-known that 
studies of a defined question are seldom replicated [1]. Such 
replication, however, is extremely valuable in assessing the 
validity of a method, which may be limited by the well-known 
Hawthorne effect, for example [2-4]. Even in cases in which 
randomization is not appropriate to the question replication is 
indicated as a matter of principle [5]. 

A recent study analyzed the complete publication history 
of the current top too education journals ranked by 5-year 
impact factor and found that only 0.13% of education articles 
were replications [1]. The authors emphasize the “importance 
of direct replications in helping education research improve 
its ability to shape education policy and practice” [1]. Such 
replication studies may be little publicized for two reasons, 
namely, that the original data from the primary study are not 
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evaluation t 7 l- The catalogue is available in German, English, 
Portuguese and Spanish, at: http://www.imbi.uni-freiburg. 
de/medinf/gmdsqc/ [7]. A modified method of evaluation has 
already been applied in a dental context [9,10]. The reliability 
values determined here for the individual factors studies 
were designated “good” with a score of 0.83-0.89 and for 
the entire questionnaire “excellent” with a score of 0.92. It is 
important to stress that all the studies which used this method 
of evaluation addressed without exception newly introduced 
implementations of e-learning programmes. To what extent 
the internal validity of each is influenced by, for example, 
the well-known Hawthorne effect is difficult to predict and 
thereby represents a serious limitation of the results. The 
fact that the e-learning resource was newly introduced and 
that participants may therefore have paid especial regard to 
it could possibly have played an important role [4]. To reduce 
the impact of this limitation and thereby make the results of 
the evaluation more generalizable, it would be sensible to re¬ 
evaluate the resource at a later date, for example, at one year 
after the initial implementation. Following from this, the focus 
of the present identical replication study was the following: 
how do dental students receive and respond to the as yet little 
researched teaching of EbD with multimedia components in 
comparison to a prior study in a different population? [6]. The 
research focus was the method of evaluation developed on the 
basis of the QCEPM. This was to be evaluated descriptively 
by direct comparison of the primary and replication studies, 
focusing on two concrete research questions: 

1) How do students in their first clinical semester of 
conservative dentistry rate the EbD e-learning resource 
in general and with respect to defined parameters? 

2) Are there differences in individual results between the 
present replication study and the results of the primary 
study? 

Material and Methods 

The e-learning resource 

The e-learning resource forms one part of the so- 
called Frankfurt Dentistry Initiative (FranZi, Frankfurter 
Zahnmedizinische Initiative). The component presented in the 
present work (“Toothache Clinic - Focus”) includes several 
interactive case vignettes. The resource was developed using 
an open-source authoring tool (WebKit Freiburg, Version 3.1 
/ beta) and is accessible without limitation online (http:// 
elearning.med.uni-frankfurt.de/spielwiese-fokus/). Many 
functions were incorporated in the 56-page resource (multiple 
choice questions drag and drop drop-down menus, draw 
functions, video clips, feedback and tips). EbD content was 
incorporated into over 57% of the total resource. EbD content 
was integrated with respect to the stipulation of the evidence- 
based medicine workgroup of the Frankfurt University which 
refers to newcomers to EbD. The initial implementation 
(primary study) of the online course “Toothache Clinic - 
Focus” was four semesters, i.e. two years, prior. 

Student group and examination intervals 

Dental students from two consecutive cohorts (summer 


Citation: Dreher S, Weberschock T, Giraki M, Uhse A, Parvini P, et al. (2017) Dental 
4(4): 066-071. DOI: http://doi.org/10.17352/2394-8418.000052 


semester 2014 and winter semester 2014/15) in their first 
clinical semester (sixth academic semester) after using the 
voluntary e-learning resource were surveyed using a written 
questionnaire. In the first meeting (c. 30 minutes long), they 
were informed about the use of the resources and told that 
they have been available for several years. None of the students 
had previous received educational content on evidence-based 
learning. A questionnaire conceived on the basis of the Ulm 
Quality Criteria Catalogue for Medical Learning Programmes 
served as the method of evaluation [8,9]. This has previously 
been used in connection with FranZi [9]. The first part of the 
questionnaire concerned general details of semester number, 
sex, age, previous marks on major state exams (Abitur, 
preliminary scientific and medical examinations) as well as 
work experience and familiarity with electronic media (Likert 
scale from 1 = “very good” to 5 = “poor”). Furthermore, special 
questions also addressed the utility of the content in patient 
treatment and the openness to implementation of EbD. The 
overall rating for the module was elicited using a scale from 1 
(very good) to 6 (insufficient) according to the German school 
marking system. Part two of the questionnaire concerned the 
evaluation of the resource in the dimensions of content and 
functional range (A), handling and technical aspects (B), and 
didactics and educational value (C) on a four-level Likert scale 
from 0 = “strongly disagree” to 3 = “strongly agree”. Students 
could use the resource for 14 days and were asked to return the 
questionnaires at a second meeting (c. 30 minutes long). 

Statistics 

The analysis was performed using PASS and NCSS (Version 
6.0.2.1. Kaysville, Utah), using using a one-way analysis of 
variance. Significant differences were determined using the 
nonparametric Kruskal-Wallis multiple-comparison Z-value 
test according to Bonferroni. Means, standard deviations (SD) 
or frequencies were also evaluated. 

After consultation with the university’s ethics committee, 
it was decided that an ethics committee vote was not necessary 
since the evaluation was anonymous. 

Results 

The descriptive results of the study are shown in tables 
1-5. The dental students gave the e-learning module in the 
present study an average score of 2.40 ± 0.66 (primary study: 
2.26 ± 0.64). For the questions on the learning programme in 
the present study, dimension A received an overall average 
score of 1.90 ± 0.57 (primary study: 1.90 ± 0.63), dimension B 
1.48 ± 1.60 (primary study: 1.55 ± 1.93) and dimension C 2.27 
± 0.67 (primary study 2.23 ± 0.79). All obtained results in the 
replication study showed no statistical significant differences 
to the primary study. 

Discussion 

The present replication study has shown that an e-learning 
resource with dental case vignettes and evidence-based 
dentistry content, which has been available for several years, 
is on average rated positively by students of dentistry in their 
first clinical semester of conservative dentistry and that the 
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users would appreciate inclusion of EbD in their course of 
study. 

The agreement of the results with those of the primary 
study points to the remarkable validity of the chosen method 
of evaluation. Currently, there exist few replication studies in 
the field of e-learning. Specifically, in the context of dentistry 
and with EbD content, no such publications are known to the 
authors [3,11]. 

Analogous to the present study, similar research in the 
field of medicine focussing on online-learning of evidence- 
based medicine (EbM) has shown that participants judge the 
courses positively and that the e-learning environment is 
covariant with clinically relevant increase in knowledge [12- 
17]. It must be noted that the foundation for the use of EbD 
content is as a rule formed from real clinical situations and that 
novices are usually involved voluntarily [18,14-16]. In a similar 
manner, the e-learning resource investigated in the present 
replication study concerned real patient cases and combined 

Table 1: General population data (* = achieved before university entrance, **= 
achieved in preliminary preclinical education, *** = achieved in final preclinical 
education. All marks ranged from 1 = very good, 2 = good, 3 = satisfactory, 4 = 
sufficient, 5 = deficient and 6 = insufficient). 



Total 

(n) 

Men 

(n) 

Women 

(n) 

Age 

(Years) 

Mean 

mark (*) 

Mean 

mark (**) 

Mean 

mark (***) 

Response 
rate (%) 

Replication 

Study 

65 

13 

52 

23,30 

1,89 

1,87 

2,49 

100,00 

Primary 

cti iH\/ 

114 

35 

79 

24,59 

2,00 

2,19 

2,58 

100,00 


study 


Table 2: General ratings. 

Item 

General ratings 

Replication 

study 

Replication 

study 



results 

results 

1 

1 rate my prior knowledge of electronic media 
as very good (1), good (2), satisfactory (3), 
sufficient (4) or deficient (5). 

2,34 ± 0,91 

2,36 ± 0,89 

2 

1 have already completed a professional 
education. 

4 of 65 

22 of 114 

3 

1 own a tablet PC. 

57 of 65 

102 of 114 

4 

On average, 1 have worked with the tutorial for 
the following no. of minutes in total. 

75,54 ± 
73,80 

67,09 ± 48,5 

5 

1 consider the module helpful for state 
examination preparation. 

53 of 65 

95 of 114 

6 

1 consider the tutorial helpful for preparing for 
the treatment of patients in clinical courses. 

55 of 65 

101 of 114 

7 

1 consider the tutorial helpful for preparing for 
the treatment of patients in the emergency unit. 

61 of 65 

108 of 114 

8 

1 would recommend the tutorial to other 

students. 

57 of 65 

107 of 114 

9 

1 would like to be able to use the e-learning 
material on a tablet PC. 

50 of 65 

89 of 114 

10 

1 find it useful to work with contents of evidence- 
based dentistry during my university studies. 

63 of 65 

105 of 114 

11 

Overall, 1 would give the module the following 
mark from 1 to 6, where 1 = 'very good', 2 = 
'good', 3 = 'satisfactory', 4 = 'sufficient', 5 = 
'deficient' and 6 = 'insufficient'. 

2,40 ± 0,66 

2,26 ± 0,64 


Table 3: Handling and technical aspects. 


Item 

A. Handling and technical aspects 

Replication study 

Primary study 



Mean ± Standard 

deviation 

Mean ± Standard 

deviation 

1 

Access of the programme is easy. 

1,95 ±0,90 

2,07 ± 0,80 

2 

System requirements and software 
limitations are clearly indicated. 

1,70 ±0,70 

1,89 ±0,70 

3 

The user interface is optically 
attractive. 

1,77 ±0,74 

1,97 ±0,71 

4 

Navigation is clear and intelligible. 

1,85 ±0,70 

2,08 ± 0,66 

5 

Access to the learning tutorial is 
uncomplicated. 

1,98 ±0,69 

2,09 ± 0,65 

6 

The programme is stable and fast. 

1,97 ±0,63 

2,1 5 ± 2,04 

7 

The font is clearly readable. 

2,05 ± 0,66 

2,22 ± 0,66 

8 

The multimedia-based elements 

work without difficulties. 

1,89 ±0,75 

2,02 ± 0,58 

9 

The media used have an acceptable 
time (upt to 15 seconds) 

2,02 ± 0,64 

2,04 ± 0,63 

10 

Text load per screen page is 
appropriate (scolling can mostly be 
avoided). 

2,03 ± 0,65 

2,09 ± 0,54 

11 

The number of multimedia-based 
elements per screen page is 
appropriate. 

2,05 ± 0,60 

2,03 ± 0,60 

12 

Visual media have an appropriate 
quality and size. 

1,92 ±0,70 

1,99 ±0,60 

13 

Pictures and figures used are of 
high quality in terms of resolution 
and detail detection. 

1,92 ±0,68 

1,83 ±0,57 

14 

Incorporated videos are of hogh 
quality with respect to clarity and 
coherence. 

1,89 ±0,68 

1,88 ±0,54 


Total 

1,92 ±0,57 

1,90 ±0,63 


the basic knowledge of EbD with orientation towards patients, 
resulting in a kind of practical knowledge. According to a 
2013 survey on courses in EbM, two of the five steps of EbM, 
namely integration of results with patients and the evaluation 
of particular merit, fell significantly behind the others and are 
thus addressed less frequently [15,16]. All steps were integrated 
into the present resource. 

Taken as a whole, the methodology of evidence-based 
dentistry (EbD) takes centre stage in the endeavour to place 
dentistry on a scientific foundation [19]. In Germany in 
particular, however, EbD is not firmly established in dentistry 
curricula. This is in stark contrast to the USA, for example, 
where the inclusion of EbD in dentistry curricula has already 
been publicised several times [20-23]. This present problem in 
Germany, that dentistry graduates show minimal competence 
in EbD at the beginning of their working life as practicing 
dentists, should in the future be solved by the medical licensing 
act for dentists currently being planned in Germany and the 
National Catalogue of Competence-based Learning Objectives 
in Dentistry containing operationalised EbM learning 
objectives which has already been adopted [24,25]. Moreover, 
a 2016 study reported that five defined parameters should be 
considered before a forthcoming implementation of EbM in 
curricula, namely 1) integrating EbM with other courses and 
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Table 4: Content and functional range. 


Item 

B. Content and functional range 

Replication study 

Primary study 



Mean ± Standard 

deviation 

Mean ± Standard 

deviation 

1 

The content was carefully 
reviewed. 

1,71 ±0,54 

1,70 ±0,67 

2 

Contact details for support and 
guidance are given. 

1,43 ±0,68 

1,60 ±0,68 

3 

Overview and directory of learning 
contents are provided. 

1,88 ±0,56 

1,91 ±0,62 

4 

Medical learning objectives are 
clearly specified. 

1,90 ±0,65 

1,97 ±0,68 

5 

Dates of content preparation and 
updating are indicated. 

1,40 ±0,83 

1,57 ±0,80 

6 

Authors and affiliations are 

named. 

1,66 ±0,75 

1,74 ±0,80 

7 

Direct contact with authors is 
encouraged (email addresses 
provided) 

1,35 ±0,79 

1,70 ±0,85 

8 

After interrupting the tutorial, the 
point of re-entry can easily be 
accessed. 

1,48 ±0,73 

1,86 ±0,67 


Total 

1,48 ±0,60 

1,55 ±1,93 


content, 2) incorporating clinical content into EbM training, 
3) EbM faculty development, 4) EbM whole-task exercises, 
and 5) longitudinal integration of EbM [26]. Based on these 
considerations, especial care was taken in the present study to 
integrate EbD content into the course of conservative dentistry 
particularly in the case of diagnosis and treatment of toothache. 
In addition, EbD content was related to realistic clinical cases. 
Faculty development was likewise apparent, since creation 
of the resource content in co-operation with the local EbM 
workgroup clearly required it. A longitudinal integration into 
several semesters of the dentistry study has not yet been 
realised, but it should be addressed in future studies. 

According to the literature, teaching of EbM content can 
be successfully achieved in various settings (teacher-centred, 
e-learning, blended learning) [13,27,28]. In the present study, 
the resource was defined as an optional means for students’ 
self-study. In the future, however, it will also be used in the 
form of blended learning. Comparable studies that used an 
identical evaluation protocol with the dimensions described 
here, albeit in other e-learning tutorials (“Toothache Clinic 
- Basic”), came to comparable conclusions with respect to 


Table 5: Didactics and educational value. 


Item 

C. Didactics and educational value 

Replication study 

Primary study 



Mean ± Standard deviation 

Mean ± Standard deviation 

1 

Extent of interactivity matches the contents. 

1,98 ±0,61 

1,96 ±0,55 

2 

Tutees receive appropriate feedback on their learning progress. 

1,97 ±0,80 

1,97 ±0,71 

3 

The tutorial permits self-assessment of the learning success. 

1,97 ±0,80 

1,97 ±0,65 

4 

Target groups and skills required for the tutorial are adequately specified. 

1,73 ±0,70 

1,77 ±0,61 

5 

The didactic concept is apparent. 

1,89 ±0,68 

2,00 ± 0,64 

6 

The multimedia-based elements used help tutees tu understand the material 
provided. 

1,94 ±0,76 

2,14 ±0,68 

7 

Real-life examples (e.g., background stories and medical history interviews) 
facilitate the learning process. 

1,71 ±0,77 

1,89 ±0,62 

8 

The tutorial improve the tutees'ability to apply differential diagnoses. 

1,87 ±0,75 

2,04 ± 0,69 

9 

The tutorial improves the tutees'competence in decision making and 
appropriate medical action. 

1,84 ± 0,70 

2,01 ± 0,67 

10 

The tutorial provied appropriate material to reach the intended learning goals. 

1,89 ±0,60 

1,87 ±0,60 

11 

The tutorial matches the requirements for theoretical (or test) as well as real- 
life situations. 

1,94 ±0,62 

2,05 ± 0,71 

12 

The tutorial indicates the structure, topics, and settings upfront. 

1,87 ±0,74 

1,89 ±0,60 

13 

Information contained in texts, pictures, and video sequences are clear and 
intelligible. 

1,85 ±0,77 

1,99 ±0,64 

14 

The contents of the tutorial are in agreement with real-life situations. 

1,92 ±0,73 

2,05 ± 0,69 

15 

The composition of multimedia-based elements is particularly appropriate to 
facilitate the learning objectives. 

1,67 ±0,73 

1,94 ±0,74 

16 

Tools (feedback, tips, drawing tools, dfrag & drop) provided are a motivating 

factor. 

1,73 ±0,83 

2,02 ± 0,69 

17 

Competitive elements (e.g., self-test, incentives to obtain a certain number of 
scores) are a motivating factor. 

1,85 ±0,72 

1,88 ±0,71 

18 

The tutorial is suitable for use during lectures. 

1,55 ±0,82 

1,61 ±0,82 

19 

The tutorial is suitable for self-study. 

1,97 ±0,87 

2,20 ± 0,67 

20 

The tutorial is suitable for the teaching in groups. 

1,71 ±0,78 

2,06 ± 0,78 

21 

The tutorial clearly adds value to the existing educational tools. 

1,68 ±0,76 

1,95 ±0,73 


Total 

2,27 ± 0,68 

2,23 ± 0,79 
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the dimension of “didactics and educational value” (2.29 
± 0.38) [8,9]. The two other dimensions were rated more 
highly, however (content and functional range: 2.32 ± 0.47 and 
handling and technical aspects: 2.47 ± 0.36) [8,9]. Here, it may 
be worth noting that these previous studies were conducted in 
2009 when the e-learning resource was completely new and 
the students were unfamiliar with it [8,9]. Three years later, 
at the time of the primary study of this investigation, the field 
of electronic media and resources seems to have developed 
greatly so that, understandably, the users gave lower ratings 
in questions of function and technical aspects. A period of 
1.5 years lay between the data of the primary and replication 
studies with EbD content which led at least to more conspicuous 
judgements for several items in the dimension “didactics and 
educational value” of the programme. For example, in the 
primary study the students gave a higher rating in regard to 
the appropriateness of the learning unit for self-study or group 
study. The users tended to take the view that the resource is 
well suited to self-study. None of the students had experience 
using the studied resource in a group or in a blended learning 
scenario, however, so that an according implementation is 
quite conceivable. 

In the study at hand, no data concerning increases in 
knowledge were analysed, though MC questions integrated 
into the resource could be consulted in future studies to 
assess increase in the users’ knowledge. The present study is 
thus concerned with a purely subjective course evaluation on 
the part of the students, to record one limitation. Likewise, 
international alignment is conceivable owing to the open 
sourcing of the resources, though they are at present only 
available in German. Plans for the translation of all FranZi 
resources into English are already underway. 

Conclusion 

In summary, the results of this clinical replication study are 
in agreement with the primary study and show that e-learning 
resources with dentistry case vignettes and integrated EbD 
content is rated positively by students of dentistry in their first 
clinical semester on conservative dentistry. 
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